Interesting approach to parallelism, what would be a concrete use case for only letting information flow one way from the sidekick to the http workers?

Usually the flow would be inverted, where a http worker offloads work to a pool of 'sidekick' workers and can optionally wait for a task to complete.

Thank you for the contribution. Interesting idea, but I'm thinking we should merge the approach with #1883. The kind of worker is the same, how they are started is but a detail.

@nicolas-grekas the Caddyfile setting should likely be per php_server, not a global setting.

@AlliBalliBaba The use case isn't task offloading (HTTP->worker), but out-of-band reconfigurability (environment->worker->HTTP). Sidekicks observe external systems (Redis Sentinel failover, secret rotation, feature flag changes, etc.) and publish updated configuration that HTTP workers pick up on their next request; with per-request consistency guaranteed via $_SERVER injection. No polling, no TTLs, no redeployment.

Task offloading (what you describe) is a valid and complementary pattern, but it solves a different problem. The non-HTTP worker foundation here could support both.

@henderkes Agreed that the underlying non-HTTP worker type overlaps with #1883. The foundation (skip HTTP startup/shutdown, immediate readiness, cooperative shutdown) is the same. The difference is the API layer and the DX goals:

• Minimal FrankenPHP config: a single sidekick_entrypoint in php_server(thanks for the idea). No need to declare individual workers in the Caddyfile. The PHP app controls which sidekicks to start via frankenphp_sidekick_start(), keeping the infrastructure config simple.

• Graceful degradability: apps should work correctly with or without FrankenPHP. The same codebase should work on FrankenPHP (with real-time reconfiguration) and on traditional setups (with static or always refreshed config).

• Nice framework integration: the sidekick_entrypoint pointing to e.g. bin/console means sidekicks are regular framework commands, making them easy to develop.

Happy to follow up with your proposals now that this is hopefully clarified.
I'm going to continue on my own a bit also :)

Great PR!

Couldn't we create a single API that covers both use case?

We try to keep the number of public symbols and config option as small as possible!

@henderkes Agreed that the underlying non-HTTP worker type overlaps with #1883. The foundation (skip HTTP startup/shutdown, immediate readiness, cooperative shutdown) is the same. The difference is the API layer and the DX goals:

Yes, that's why I'd like to unify the two API's and background implementations into one. Unfortunately the first task worker attempt didn't make it into main, but perhaps @AlliBalliBaba can use his experience with the previous PR to influence this one. I'd be more in favour of a general API, than a specific sidecar one.

The PHP-side API has been significantly reworked since the initial iteration: I replaced $_SERVER injection with explicit get_vars/set_vars protocol.

The old design used frankenphp_set_server_var() to inject values into $_SERVER implicitly. The new design uses an explicit request/response model:

• frankenphp_sidekick_set_vars(array $vars): called from the sidekick to publish a complete snapshot atomically
• frankenphp_sidekick_get_vars(string|array $name, float $timeout = 30.0): array: called from HTTP workers to read the latest vars

Key improvements:

• No race condition on startup: get_vars blocks until the sidekick has called set_vars. The old design had a race where HTTP requests could arrive before the sidekick had published its values.
• Strict context enforcement: set_vars and should_stop throw RuntimeException if called from a non-sidekick context.
• Atomic snapshots: set_vars replaces all vars at once. No partial state possible
• Parallel start: get_vars(['redis-watcher', 'feature-flags']) starts all sidekicks concurrently, waits for all, returns vars keyed by name.
• Works in both worker and non-worker mode: get_vars works from any PHP script served by php_server, not just from frankenphp_handle_request() workers.

Other changes:

• sidekick_entrypoint moved from global frankenphp block to per-php_server (as @henderkes suggested)
• Removed the $argv parameter: the sidekick name is the command, passed as $_SERVER['argv'][1]
• set_vars is restricted to sidekick context only (throws if called from HTTP workers)
• get_vars accepts string|array: when given an array, all sidekicks start in parallel
• Atomic snapshots: set_vars replaces all vars at once, no partial state
• Binary-safe values (null bytes, UTF-8)

Thanks @dunglas and @henderkes for the feedback. I share the goal of keeping the API surface minimal.

Thinking about it more, the current API is actually quite small and already general:

• 1 Caddyfile setting: sidekick_entrypoint (per php_server)
• 3 PHP functions: get_vars, set_vars, should_stop

The name "sidekick" works as a generic concept: a helper running alongside. The current set_vars/get_vars protocol covers the config-publishing use case. For task offloading (HTTP->worker) later, the same sidekick infrastructure could support:

• frankenphp_sidekick_send_task(string $name, mixed $payload): mixed
• frankenphp_sidekick_receive_task(): mixed

Same worker type, same sidekick_entrypoint, same should_stop(). Just a different communication pattern added on top. No new config, no new worker type.

So the path would be:

1. This PR: sidekicks with set_vars/get_vars (config publishing)
2. Future PR: add send_task/receive_task (task offloading), reusing the same non-HTTP worker foundation

The foundation (non-HTTP threads, cooperative shutdown, crash recovery, per-php_server scoping) is shared. Only the communication primitives differ.

WDYT?

I think the failures are unrelated - a cache reset would be needed. Any help on this topic?

Hmm, it seems they are on some versions, for example here: https://github.com/php/frankenphp/actions/runs/23192689128/job/67392820942?pr=2287#step:10:3614

For the cache, I'm not aware of a Github feature that allow to clear everything unfortunately 🙁

Note that I would love to have workers available for CLI scripts. Something that's entirely skipped for now.
I'd leave this for a future PR.

Isn't calling set_vars the ready-state we already have?

You're absolutely right. The point about the stream logic being quite low level for users still stands, though.

Note that I would love to have workers available for CLI scripts. Something that's entirely skipped for now.
I'd leave this for a future PR.

php/php-src#21354

Should wait for that first. The current solution with an emulated cli through the embed sapi is a bit hacky and causes issues. I'd prefer not to add much to it until we get a proper cli sapi embedded.

after discussion with @nicolas-grekas today at symfony live, i tried out if we could offer a polyfill for this functionality. it is still very rough, but it shows the basic PoC by writing to a filesystem: https://github.com/dbu/shared-state-polyfill

i'd be happy to evolve on it a bit, after getting input - though probably its best to wait until this PR is merged or at least the contract is sure to be final.

(i don't want to hijack this thread, please open issues in my PoC repository to discuss the polyfill. @nicolas-grekas if you could review the basic thing before the frankenphp changes get merged, we can do a sanity check if we maybe should change something of the pattern to make it polyfill friendly. the notification stream is a bit funny for the polyfill as it would be unnecessary as each worker is its own process that receives the signals directly. and the name of the background worker being implicit makes polyfill a bit awkward, but is great for a smooth experience with frankenphp.
and last, do we really want to prefix the methods with frankenphp when the concept is applicable in other context as well?)

Would be nice if someone with fresh eyes could look over this PR. Short summary from my side:

• the PR is still very big, there's probably a lot of room for simplification
• the API is very exotic. Not necessarily bad, but it would be nice if we could have more general abstractions or a similar implementation for orientation somewhere else.
• Still feels a bit messy to allow starting background workers at runtime. Is the main use case that libraries can start workers without the user configuring them?
• Doing shutdown via stream is very clunky
• Would be nice to have clean sync shutdown with signals, not sure this is possible rn
• no integration caddy_tests
• copying across threads is still very limited in what you can copy (something to be aware of in the future)

Another reason I dislike streams is that the polling api from (this RFC) might already be in PHP 8.6.

Thanks for the review. Addressing each point:

PR size / simplification: I removed dead methods, moved scope management to the frankenphp package (NextBackgroundWorkerScope()), added a Caddy integration test (TestBackgroundWorkerCaddy). Happy to simplify further if you see specific spots.

API: set_vars/get_vars/get_signaling_stream - three functions, one Caddyfile directive. I'm not sure what a simpler abstraction would look like for cross-thread shared state with cooperative shutdown. The cooperative part is critical to me: background workers have fundamentally different execution models than HTTP ones.

Runtime worker starting: The main use case is libraries that ship their own background workers (e.g., a Redis Sentinel watcher package) without requiring users to list each one in the Caddyfile. The catch-all worker { background } enables this. Named workers with num 1 cover the explicit case. Both patterns are opt-in from the Caddyfile - PHP can't start workers that aren't declared.

Cross-thread copying: Intentionally limited to scalars/arrays/enums - this enables immutable array zero-copy, interned string optimization, and the === skip in set_vars. Objects would require serialize/unserialize which breaks SRP (foreign exceptions from the copy layer). For complex objects, Symfony's DeepCloner (symfony/symfony#63695) converts any serializable value to an array - perfect fit for set_vars. Explicit serialize() remains an option.

Stream-based shutdown: I explored signal-based shutdown extensively (pthread_kill, timer_create, CancelSynchronousIo). Go's signal trampoline makes this unviable cross-platform - documented in the PR.

On the "clunky" concern, it's worth noting that the upcoming PHP 8.6 poll API RFC validates this direction: it introduces Poll as a modern replacement for stream_select, with platform-optimized backends (epoll, kqueue, WSAPoll). Our signaling stream is forward-compatible: Poll::addReadable accepts stream resources directly. Code written today with stream_select will work on 8.6 with Poll - no API change needed on the FrankenPHP side. The stream approach isn't a stopgap, it's the foundation that the poll API builds on.

Example of forward compatibility - same code, both worlds:

// Works today (PHP 8.2+)
$stream = frankenphp_worker_get_signaling_stream();
stream_select([$stream], [], [], 5);
$signal = fgets($stream);

// Works on PHP 8.6+ with zero changes to FrankenPHP
$poll = new Poll();
$poll->addReadable($stream, fn() => match (fgets($stream)) {
    "stop\n" => false,
    "task\n" => handleTask(), // anticipating from the task-based API in #2319 
});
$poll->addTimer(5.0, fn() => frankenphp_worker_set_vars(refreshConfig()));
$poll->run();

Same $stream, no new FrankenPHP API needed. And if the poll RFC lands with a SignalHandle type, we could later add a higher-level alternative that hides the string parsing entirely (to be confirmed it's worth it in the future):

// Possible future API (PHP 8.6+, if SignalHandle lands)
$poll = new Poll();
$poll->addHandle(frankenphp_worker_get_poll_handle(), fn(string $signal) => match ($signal) {
    'stop' => false,
    'task' => handleTask(),
});
$poll->addTimer(5.0, fn() => frankenphp_worker_set_vars(refreshConfig()));
$poll->run();

The stream version stays for BC, same underlying pipe, different wrapper.

For simple cases today, the userland background_worker_should_stop() helper wraps the stream_select boilerplate into a one-liner.

Protocol update: instead of using "stop\n" for signaling cooperative shutdown, closing the stream does the job. This means replacing what was "stop\n" === fgets($h) before by false === fgets($h). Cleaner.

I'm now leaning toward renaming functions:

• frankenphp_get_worker_handle() instead of frankenphp_worker_get_signaling_stream()
• frankenphp_get_worker_vars() instead of frankenphp_worker_get_vars()
• frankenphp_set_worker_vars() instead of frankenphp_worker_set_vars()

WDYT?

Or possibly (suggested at SymfonyLive Paris), using background instead of worker:

• frankenphp_get_background_handle()
• frankenphp_get_background_vars()
• frankenphp_set_background_vars()

Use reactions to express your preference? (👎 on the previous to prefer the frankenphp_worker_*() style)

Another reason I dislike streams is that the polling api from (this RFC) might already be in PHP 8.6.

@AlliBalliBaba If anything that's a positive because it's built upon streams and will just make the userland handling cleaner (which is my main objection for the stream api).

Protocol update: instead of using "stop\n" for signaling cooperative shutdown, closing the stream does the job. This means replacing what was "stop\n" === fgets($h) before by false === fgets($h). Cleaner.

Much better!

frankenphp_get_worker_handle() instead of frankenphp_worker_get_signaling_stream()

I like it better than get_signaling_stream. Especially because:

frankenphp_get_worker_vars() instead of frankenphp_worker_get_vars()
frankenphp_set_worker_vars() instead of frankenphp_worker_set_vars()

Is there actually a reason to not make this frankenphp_get_vars and frankenphp_set_vars? It's less specific, but that would allow expanding it to regular threads or requests later. If it's specific to a worker perhaps we could make that an (optional?) argument that accepts a worker handle?

Thanks for the push in that direction :)

@dunglas asked me why only arrays as "vars"? Suggesting we should accept strings and other scalars.

This made me change the API this way. Still 3 functions:

• frankenphp_set_worker_state()
• frankenphp_get_worker_state()
• frankenphp_get_worker_handle()

About an HTTP-worker state, the purpose would be to expose a thread-safe in-memory KV store to sync them?
What about defining a convention like: http-workers in the empty string $name?